Method and apparatus for on-chip per-pixel pseudo-random time coded exposure

ABSTRACT

Conventional methods for imaging transient targets are constrained by a trade-off between resolution and frame rate, and transient targets moving faster than the detector frame typically result in image blurring. Imagers using digital-pixel focal plane arrays (“DFPAs”) have on-chip global pixel operation capability for extracting a single transient-feature (i.e., single-frequency discrimination) in a snapshot that depends on the number of counters implemented per pixel. However, these DFPA systems are not capable of multi-target and multi-frequency discrimination. Imagers described herein achieve multi-target transient signature discrimination orders of magnitude faster than the readout frame rate using in-pixel electronic shuttering with a known time-encoded modulation. Three-dimensional (x,y,t) data cube reconstruction is performed using compressive sensing algorithms.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/678,439, now U.S. Pat. No. 10,250,831, which was filed on Aug. 16,2017, as a continuation of U.S. application Ser. No. 14/789,601, nowU.S. Pat. No. 9,743,024, which was filed on Jul. 1, 2015. Each of theseapplications is incorporated herein by reference in its entirety.

GOVERNMENT SUPPORT

This invention was made with government support under Contract No.FA8721-05-0002 awarded by the U.S. Air Force. The government has certainrights in the invention.

BACKGROUND

Focal plane arrays (FPAs) are two-dimensional arrays of photodetectorsdisposed in the focal plane of a lens for applications such as imaging,spectrometry, lidar, and wave-front sensing. Conventional FPAs typicallyprovide analog readout, with the analog signals generated at the pixellevel converted to digital signals by analog-to-digital converters(ADCs) that are external to the FPA. The converted signals can then beprocessed according to the demands of a particular application. Specificanalog designs can target, and possibly achieve, one or more designrequirements, but fail when simultaneously targeting the most aggressivedesign parameters for imaging applications, such as transient targetimaging and wide-area surveillance.

Wide-area imaging with commercial off-the-shelf (COTS) FPA detectorstrade frame rate for spatial resolution. As a result, fast movingtargets are recorded as streaks moving across a field-of-view. Thesetwo-dimensional (2D) streaked images lack temporal history and providedegraded spatial images. Conventional digital-pixel focal plane arrays(“DFPAs”) have on-chip global capability for single transient-featureextraction in a snapshot, but are not capable of multi-target andmulti-frequency discrimination. Persistent and wide-area surveillanceapplications can therefore benefit from techniques that enable searchand track of fleeting targets while maintaining a wide field-of-view.

SUMMARY

Embodiments of the present disclosure include a focal plane imagingapparatus and method for on-chip per-pixel space-time modulation. Anexample focal plane imaging apparatus includes: (1) a plurality ofphotodetectors, with a first photodetector to convert a first portion oflight that is scattered and/or reflected from a scene into a firstanalog signal and a second photodetector to convert a second portion oflight that is scattered and/or reflected from the scene into a secondanalog signal; (2) a plurality of analog to digital converters (ADCs)with a first ADC electrically coupled to the first photodetector andconfigured to convert the first analog signal into a first digitalsignal, and a second ADC electrically coupled to the secondphotodetector and configured to convert the second analog signal into asecond digital signal; (3) a plurality of digital registers with a firstdigital register electrically coupled to the first ADC and configured tostore a first digital number representing the first digital signal, anda second digital register electrically coupled to the second ADC andconfigured to store a second digital number representing the seconddigital signal; and (4) a distributed control pattern generator operablycoupled to the plurality of ADCs and/or the plurality of digitalregisters and configured to modulate, at a rate faster than a readoutrate of the plurality of digital registers, storage of the first digitalnumber with a first pseudo-random modulation and to modulate, at a ratefaster than a readout rate of the plurality of digital registers,storage of the second digital number with a second pseudo-randommodulation so as to control spatial correlation of the first digitalnumber with the second digital number.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the inventive subject matter disclosed herein. In particular, allcombinations of claimed subject matter appearing at the end of thisdisclosure are contemplated as being part of the inventive subjectmatter disclosed herein. It should also be appreciated that terminologyexplicitly employed herein that also may appear in any disclosureincorporated by reference should be accorded a meaning most consistentwith the particular concepts disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The skilled artisan will understand that the drawings primarily are forillustrative purposes and are not intended to limit the scope of theinventive subject matter described herein. The drawings are notnecessarily to scale; in some instances, various aspects of theinventive subject matter disclosed herein may be shown exaggerated orenlarged in the drawings to facilitate an understanding of differentfeatures. In the drawings, like reference characters generally refer tolike features (e.g., functionally similar and/or structurally similarelements).

FIG. 1A shows a focal plane imaging apparatus, including a plurality ofphotodetectors, a plurality of analog-to-digital converters (ADCs), aplurality of digital registers, and circuitry for in-pixel modulation,according to some embodiments.

FIG. 1B is a schematic showing a pixel circuit having multiple counters,suitable for use in the focal plane imaging apparatus of FIG. 1A.

FIG. 1C is a diagram showing the generation and storage of a compressivedata set for a pixel having a set of multiple counters, suitable for usein the focal plane imaging apparatus of FIG. 1A.

FIG. 1D shows a focal plane imaging apparatus, including photodetectors,ADCs, digital registers, and one or more control registers incommunication with off-chip electronics, according to some embodiments.

FIG. 1E shows a focal plane imaging apparatus, including photodetectors,ADCs, digital registers, and a control register ring buffer incommunication with off-chip electronics, according to some embodiments.

FIG. 1F shows a focal plane imaging apparatus, including photodetectors,ADCs), a plurality of digital registers, and a configuration registerand comparator in communication with off-chip electronics, according tosome embodiments.

FIG. 1G shows a focal plane imaging apparatus, including a plurality ofphotodetectors, a plurality of analog-to-digital converters (ADCs), aplurality of digital registers, and a linear feedback shift register(LFSR) and comparator in communication with off-chip electronics,according to some embodiments.

FIG. 1H shows a focal plane imaging apparatus, including a plurality ofphotodetectors, a plurality of analog-to-digital converters (ADCs), aplurality of digital registers, and a linear feedback shift register incommunication with off-chip electronics, according to some embodiments.

FIG. 1I shows a focal plane imaging apparatus, including a plurality ofphotodetectors, a plurality of analog-to-digital converters (ADCs), aplurality of digital registers, and distributed control patterngeneration circuitry for in-pixel modulation, according to someembodiments.

FIG. 2A shows a distributed control pattern generator circuit thatincludes interconnected control registers in each pixel of a focal planeimaging apparatus, according to some embodiments.

FIG. 2B is a schematic illustration of the distributed control patterngenerator of FIG. 2A, according to some embodiments.

FIG. 2C is a schematic illustration of the distributed control patterngenerator of FIG. 2A, including an exclusive OR (“XOR”) gate, accordingto some embodiments.

FIG. 2D is a schematic illustration of the distributed control patterngenerator of FIG. 2A, showing direct inter-pixel transfer connections,according to some embodiments.

FIG. 2E is a schematic illustration of the distributed control patterngenerator of FIG. 2A, showing XOR-gate transfer connections, accordingto some embodiments.

FIG. 3 is a schematic showing a pixel circuit, with a pulse modulatorand a bidirectional counter, suitable for use in the focal plane imagingapparatus of FIGS. 1A and 1D-1I.

FIG. 4A is a schematic showing a bidirectional (up/down) countersuitable for use in the focal plane imaging apparatus of FIGS. 1A and1D-1I.

FIG. 4B is a schematic showing a negate counter suitable for use in thefocal plane imaging apparatus of FIGS. 1A and 1D-1I.

FIG. 5 is a schematic showing a pixel circuit with local ADC gainconfigurability suitable for use in the focal plane imaging apparatus ofFIGS. 1A and 1D-1I.

FIG. 6 is a schematic showing a counter/register block with a clock, amode selection bus, and an enable pin suitable for use in the focalplane imaging apparatus of FIGS. 1A and 1D-1I.

FIG. 7 is a schematic showing a pixel circuit with an ADC, logic, and adigital register suitable for use in the focal plane imaging apparatusof FIGS. 1A and 1D-1I.

FIG. 8 is a schematic showing a pixel circuit with a preamplifier, apulse frequency modulator, local control logic enabling a pseudorandommodulation sequence, and a bidirectional counter suitable for use in thefocal plane imaging apparatus of FIGS. 1A and 1D-1I.

FIG. 9 is a schematic showing a digital logic circuit for two-bit pixelgroup assignment implementation suitable for use in the focal planeimaging apparatus of FIGS. 1A and 1D-1I.

FIG. 10 is a schematic showing a linear feedback shift register (LFSR)applying pseudorandom pixel group assignment to a plurality of pixels,according to some embodiments.

FIG. 11 is a schematic showing a LFSR as a pseudorandom number generatorfor pixel group assignments, according to some embodiments.

FIG. 12 is a rendering showing a 16×16 sub-region of a detector arrayand a locally applied LFSR bit mapping for a 4×4 region of interest,according to some embodiments.

FIGS. 13A and 13B are space-time per-pixel modulation correlationmatrices for pixel-group based encoding, according to some embodiments.

FIG. 13C is a space-time per-pixel modulation correlation matrix forlocal LFSR pseudorandom encoding, according to some embodiments.

FIG. 13D is a space-time per-pixel modulation correlation matrix forencoding using four pixel groups, according to some embodiments.

FIG. 13E is a space-time per-pixel modulation correlation matrix for“ideal” LFSR modulation with off-diagonal mask correlation, according tosome embodiments.

FIG. 13F is a space-time per-pixel modulation correlation matrix forencoding using four pixel groups, according to some embodiments.

FIGS. 13G and 13H are space-time per-pixel modulation correlationmatrices for distributed LFSR modulation (at two different scales),according to some embodiments.

FIG. 14 is a flow chart that illustrates a process for focal planeimaging using a digital focal plane array (DFPA), according to someembodiments.

FIG. 15 shows a series of images that have been reconstructed usinglocal, in-pixel pseudorandom time-varying modulation, according to someembodiments.

FIG. 16A is a spatially extended baseline image of three transienttargets (bullets), according to some embodiments.

FIG. 16B is an example frame of a spatially randomized temporallymodulated coded aperture used for imaging the baseline image of FIG.16A.

FIG. 16C is a modulated pixel output image of the three transienttargets of FIG. 16A, taken at a base frame rate.

FIG. 16D is a series of images comprising reconstructed data yielding 15time slices per frame acquired while imaging the three transient targetsof FIG. 16A using the spatial modulation of FIG. 16B and temporalmodulation via an LFSR as shown in FIG. 9.

FIG. 16E is a plot of peak signal-to-noise ratio (“PSNR”) valuescomparing baseline (x,y,t) target data of FIG. 16A with data-cubeestimates versus sub-frame number, according to some embodiments.

FIG. 17A is a simulated image of the three transient targets of FIG. 16Arecorded using conventional modulation at a base readout frame rate,according to some embodiments.

FIG. 17B is a simulated image of the three transient targets of FIG. 16Ausing on-chip per-pixel modulation at a base readout frame rate,according to some embodiments.

FIG. 17C is a simulated image of the three transient targets of FIG. 16Ausing on-chip global modulation at a base readout frame rate, accordingto some embodiments.

FIG. 18A is an annotated image frame showing pixel group assignments,according to some embodiments.

FIGS. 18B and 18C are image frames created using context-relevant signalprocessing within each group shown in FIG. 18A.

FIG. 19 is an annotated image frame showing pixel group assignments,according to some embodiments.

DETAILED DESCRIPTION

“Transient target imaging” refers to imaging of fast-moving and/orfast-changing subjects such as projectiles. Existing transient targetimaging using commercial off-the-shelf (“COTS”) detectors imposes atrade-off between resolution and frame rate, particularly when imaging alarge area. Transient targets moving faster than the detector frame ratemanifest themselves as streaks or “blurring” when detected by slow-speeddetectors. Temporal space-time filtering operations have previously beenimplemented via active optical image plane coding in the fields ofcomputational imaging, compressive sensing, and coded-aperture imaging.But active illumination in such contexts involves structuredillumination employing mechanical shutters such as digital micro-mirrordevices (“DMDs”). Challenges associated with mechanical shutters alongthe optical axis include speed limitations for in-pixel binaryoperations, reflectivity losses at the DMD array, mechanicalrepeatability of the actuators over time, and inherent light loss due toa 1,0 mask realization. Other approaches, e.g., in the computer visioncommunity, use a random pixel-based coding strategy with a highframe-rate amplitude modulator affecting polarization of the incidentlight (e.g., Liquid Crystal on Silicon—LCOS). Imagers usingdigital-pixel focal plane arrays (“DFPAs”) have on-chip global pixeloperation capability for extracting a single transient-feature (e.g.,single-frequency discrimination) in a snapshot that depends on thenumber of counters implemented per pixel. However, these DFPA systemsare generally not capable of multi-target and multi-frequencydiscrimination.

Embodiments of the present disclosure extend DFPA technology to supporton-chip temporal discrimination of diverse and dynamic targets orders ofmagnitude faster than the readout frame rate, thereby overcoming theconventional tradeoff between resolution and frame rate. On-chipper-pixel space-time modulation is applied to a DFPA architecture toachieve multi-target transient signature discrimination, or “temporalsuper-resolution.” A hardware in-pixel approach (a per-pixel shutterinstead of a global shutter) is employed, which overcomes thelimitations associated with mechanical apertures and allows foraperiodic and periodic multi-target transient feature discrimination ofresolved and unresolved targets. A pixel circuit receives inputphotocurrent from a photodetector, and using an analog-to-digitalconverter, creates a digital number of that photocurrent in a digitalregister. A per-pixel temporal filter (e.g., electronic shutter) is usedinstead of a global filtering operation across the array.

In some embodiments, a time-encoded streak is embedded in the image suchthat, when coupled with a known time-encoded modulation, the transientsignature can be recovered from the image with novel processingtechniques. The modulation scheme at the image plane employs a binary(+1, −1), per-pixel encoding and thus is “passive”—it has no movingparts or loss in flux from the target (i.e., there is no need for ashuttering aperture to be placed along the optical axis). In adual-counter pixel, encoding can be selectively applied to a singlecounter, while preserving a pristine raw integrated image in the othercounter. In subsequent off-chip processing, reconstruction algorithmsusing compressive sensing techniques (e.g., total variationregularization) can be applied to estimate a three-dimensional (“3D”)(x,y,t) data cube, in which a single readout frame provides manysub-frame time slices. In this case, the three dimensions, “(x,y,t),”are two spatial dimensions (e.g., “x” and “y” or “r” and “θ”) and time“t.”

Embodiments of on-chip, per-pixel modulation described herein canprovide numerous advantages. For example, the on-chip capability fortemporal modulation faster than the detector readout frame ratemitigates temporal registration issues, thereby reducing the temporalblurring in the image. The on-chip capability also enables backgroundsuppression during low-signal transient target tracking anddiscrimination by isolating background frequencies from signalfrequencies using per-pixel pseudorandom demodulation approaches.Additionally, combining a time-averaged series of per-pixel modulationand nonlinear signal inversion results in improved transient featureextraction for resolved and unresolved targets. Focal plane imagingapparatuses described herein can support low-latency closed-loopoperation, reduce the raw data volume, support high resolution over awide field of view, and allow high-rate multiframe averaging. Focalplane imaging apparatuses described herein are suitable for use in manyapplications, including image segmentation for active contour-basedtracking, local correlation tracking, optical flow measurement,observation of local temporal features, local clutter rejection, androtational motion compensation.

Implementation of Focal Plane Imaging Apparatus

For the examples which follow, a pseudorandom modulation sequence isused to modulate the pixel, but other forms of spatially and temporallyvarying modulation can be applied using the same techniques. Forexample, a structured modulation pattern containing a plurality ofspatial or temporal frequency components could be used in lieu of apseudorandom pattern.

FIG. 1A shows a block diagram of an exemplary focal plane imagingapparatus 100 that includes an off-chip processor 120 coupled to adigital read-out integrated circuit (DROIC) 130. The focal plane imagingapparatus 100 may also be referred to as a compressive imaging apparatus(CIA) because it leverages space-time logic at the local and globalpixel level for transient target imaging as described in greater detailbelow. The focal plane imaging apparatus 100 provides in-pixel local andglobal shuttering as well as compressive temporal imaging that enablesfull 3D data-cube estimation from a single temporally-encoded 2D objectprojection. Compressive sensing algorithms enable sampling below theNyquist rate for adequate signal recovery when the target sparsitymatrix and the random projection matrix satisfy the restricted isometryproperty (RIP). In theory, RIP can be derived for a wide variety ofimplementation matrices, but in practice the metric of choice is ameasure of incoherence between the measurement matrix and objectsparsity transformation matrix. Thus, the focal plane imaging apparatus100 combines the strengths associated with signal recovery fromsub-Nyquist measurements and novel DROIC design for temporalsuper-resolution.

The DROIC 130 comprises three main functional blocks: an analog frontend, a pixel digital back end and the readout circuitry. Thesefunctional blocks are implemented in an array of pixels 102 connected toone another via control/transfer lines 104. As illustrated in FIG. 1I,the control/transfer lines 104 may be an interconnection network 104 abetween in-pixel control pattern generation circuitry 199 thus forming adistributed control pattern generation circuit, or the control/transferlines may be other connections 104 b that serve other purposes. Eachpixel 102 includes a respective photodetector 101, a respectivepre-amplifier 103, a respective analog-to-digital converter (“ADC”) 105(e.g., a pulse frequency converter), one or more respective digitalregisters 107 (e.g., a counter, a bidirectional counter, a negatecounter, etc.), and local control logic 113 that forms part of adistributed control pattern generation circuit that generates apseudorandom modulation sequence. This modulation sequence can beoperably coupled to either the transduction or the digitization process.For the apparatus 100 illustrated in FIG. 1A, the modulation is coupledthrough an interface with the counter 107 through control of an enablepin and count direction selection pin. In alternate embodiments, thelocal control logic 113 could be coupled to other pixel circuitcomponents, such as the pulse frequency modulator 105, preamplifier 103,or photodiode 101, to achieve a similar effect. As shown in FIG. 1A, thelocal control logic 113 can also be operably coupled to other pixels 102(e.g., neighboring pixels, optionally in one or more orthogonaldirections within the array) in the DROIC, e.g., to receiveconfiguration information via an on-chip distributed control patterngenerator (discussed further below). As shown in FIG. 1A, the focalplane imaging apparatus 100 may also include a map register 140 that canbe implemented within the pixel and either populated with data generatedon-chip (i.e., on the DROIC 130) or from data originating from eitheroff chip or from the periphery, e.g. from an off-chip map memory 198.

The photodetectors 101 can be fabricated as a separate array (not shown)that is hybridized or otherwise coupled to the DROIC 130. For example,the photodetectors 101 may be implemented as a commercial-off-the-shelf(COTS) long-wave infrared (LWIR) InGaAs detector array that ishybridized to a 256×256 pixel DROIC with a 30 μm pixel pitch via indiumbump bonding. The photodetector array can also be implemented as ashort-wave infrared (SWIR) InGaAs camera capable of supporting a 1 kHzglobal frame rate and a 100 MHz on-chip per-pixel shutter rate fortime-varying, pseudo-random, binary-valued pixel modulation.

The photodetectors 101 can also be implemented as detectors sensitive tovisible light. For instance, the photodetectors 101 can be formed as a32×32 detector array with a 30-μm pixel pitch. For a photodiode activearea of 2.17 μm2, the pixel fill is factor of about 0.24 percent. Thisphotodiode array can be mounted in a 120-pin pin-grid array (PGA)package and wire bonded to a DROIC 130 fabricated using 45 nm IBMsilicon-on-insulator (SOI) technology to yield an apparatus 100 capableof a 6 kHz global frame rate and a 50 MHz on-chip global shutter rate.

In operation, each photodetector 101 converts incident photons, whichmay be scattered and/or reflected from a scene, into an analog signal(e.g., a photocurrent), thereby generating an analog representation ofthe incident light. Depending on the specific implementation, thephotodetectors 101 may be configured to sense visible light, short-waveinfrared (SWIR) light, or long-wave infrared light (LWIR).Alternatively, the photodetectors 101 may be replaced by or augmentedwith another type of imaging array, such as a bolometer, Terahertzimaging array, or acoustic sensor array, to sense radiation atwavelengths/energies outside the optical band.

The output of each photodetector 101 is coupled to an optionalpreamplifier 103, which amplifies the analog signal prior toanalog-to-digital conversion. The preamplifier stages 103 depicted inFIG. 1A are not necessary, but yield improved sensitivity in the SWIRregion of the electromagnetic spectrum. The preamplifier stages 103 canyield a lower least-significant-bit of tens of electrons, depending onthe gain mode setting, which reduces the quantization noise for SWIRimaging and other applications.

Each preamplifier 103 is electrically coupled to a corresponding ADC105, which converts the pre-amplified analog signal from thepreamplifier 103 into a digital signal that represents the photocurrentor other analog signal generated by the photodetector 101 due toincident light). Each ADC 105 can be implemented as a sigma-delta ADCthat converts the pre-amplified analog signal, such as a current orvoltage, into a stream of output pulses proportional to the amplitude ofthe pre-amplified analog signal (e.g., input photocurrent) as wellunderstood in the art. For instance, the ADC 105 may be a sigma-deltaADC capable of recording up to 28 bits of in-pixel digital output. OtherADC implementations are also possible.

Each digital register/counter 107 is electrically coupled to acorresponding ADC 105 and stores a respective digital numberrepresenting the digital signal received from the corresponding ADC 105.For instance, the counter 107 may count the pulses in the pulse streamemitted by a sigma-delta ADC. As shown in FIG. 1A, each counter 107includes a clock pin, an enable pin, and a counter direction (“UP/DOWN”)pin. The local control logic 113 may modulate the count (digital number)stored in the counter 107 by applying an appropriate signal to the clockpin, enable pin, and/or counter direction pin.

In some configurations, each pixel 102 may include multiple digitalregisters. For example, a given pixel 102 may include two digitalregisters, each of which is operably coupled to the output of thecorresponding ADC 105. The first digital register may collect themodulated image data, as in FIG. 1A, and the second digital register maycollect an unmodulated image of the scene. This makes it possible tocollect both a raw image and a processed image simultaneously usingon-chip components.

Alternatively, or in addition, each pixel 102 in the DROIC 130 mayinclude multiple counters. For example, the DROIC 130 may include eight8-bit counters per pixel, each of which can operate at a differentintegration time set by the local control logic 113 or processor 120.These integration times may be independent of each other or linked,e.g., in a specific ratio, and may be up to the detector readout period(reciprocal of the detector readout rate), thereby allowing the user toleverage on-chip global temporal multiplexing capabilities.

FIG. 1B shows a schematic of a pixel 102 a including a photodiode 101(electrically connected to positive power supply voltage V_(DD)), aMOSFET 108 (electrically connected/biased to voltage V_(GLOW) to biasthe photodetector, an integration capacitor 109, a reset switch 110(electrically connected to a comparator circuit reset voltageV_(RSTCBN)), and an operational amplifier (“OP/AMP”) comparator 111having a transfer function V_(out)=g_(m)(V_(int)−V_(m)). As shown inFIG. 1B, rather than a single counter, as shown in pixel 102 of FIG. 1A,pixel 102 a includes eight independent counters 107 (though otherquantities of counters per pixel are also contemplated), each controlledby a respective enable signal (collectively, enable signals 150). Eachenable signal 150 may be driven by its own independent modulationpattern, thus allowing for recording of multiple independent temporallyencoded values. As shown in FIG. 1B, each of the multiple countersincludes its own respective control gate and associated logic. The ANDgates shown implementing the enables may be any gating logic. Thevarious approaches to modulating counters described elsewhere in thisdocument can be applied to each of the individual counters shown in FIG.1B. So, for example, instead of an enable for each counter as shownhere, an up/down count direction select for each counter can be used.

FIG. 1C shows an embodiment in which multiple in-pixel counters 107record the results of modulating the input detected signal usingmultiple modulation functions 151 (which may represent the modulation ofenable signals 150, discussed with reference to FIG. 1B) via globalcontrol logic 114. This generates a set of compressive data 153 in eachpixel's set of counters. This data may be used by a matched filter orBayesian processor 155 to generate a detection map 156 indicating thoseparts of the scene with high or low correlation to signatures in alibrary of signatures stored in a database 154. In some embodiments,signatures in the library of signatures of database 154 are derived insuch a way as to correspond to the control patterns generated by thedistributed control pattern generator (which, for example, can includelocal control logic, such as that shown in FIG. 1A at 113). In someembodiments, the Bayesian processor 155 and the database 154 arefunctionally part of a processor block (e.g., processor 120 of FIG. 1A).The processor can be off-chip, on-chip, or partially on and off-chip.

FIG. 1A also shows that each pixel 102 includes local control logic 113that is operably coupled to the ADC 105 and/or the digital register 107in the same pixel 102. The in-pixel local control logic 113 can also beoperably coupled to one or more neighboring pixels within the DROIC 130,as part of a distributed control pattern generator circuit, discussedfurther below. The in-pixel local control logic 113 enables apseudo-random pixel code to be pre-loaded at the pixel level andtemporally regrouped and negated. In operation, the local control logic113 modulates the digital register(s) 107 in the same pixel with thepseudo-random pixel code, also called an aperture code or shutter code,at a rate faster than the readout rate of the digital register(s) 107.For instance, the local control logic 113 may apply the pseudo-randommodulation to a given register's enable pin, count direction pin, clockpin, and/or mode selection bus. The pseudo-random modulation causes eachregister 107 to store a respective digital number that represents apseudo-randomly modulated measure of the aggregate photon fluxintegrated during the register readout period.

Different registers 107 in the DROIC 130 may be modulated with differentpseudo-random aperture codes in order to control the spatial correlationamong the pixels 102 in the array. For example, the pseudo-randomaperture codes may be selected or generated such that they areuncorrelated in order to facilitate the use of compressive sensing imageprocessing techniques as described in greater detail below. Thepseudo-random modulation can be generated by a temporal modulator thatis coupled to the ADC 105 and/or the register 107, a value stored inmemory (e.g., map register 140), or a configuration bit of the register107. The pseudo-random modulation may also be based on a peripheralselection signal that originates outside of the DROIC 130 (e.g., asignal from the processor 120, a signal determined by a computer orother device which uses the resulting imagery, etc.).

In some cases, the pixels 102 may be divided into different groups, witha different pseudo-modulation applied to each group and the groupassignments stored in the map register 140, which can be implemented asa 14-bit register. The pixels 102 can be assigned to groupspseudo-randomly via a linear shift feedback register (LFSR) using theleast-significant bit and most-significant bit of the map register 140.Note that LFSRs can also be utilized to produce pseudo-random sequencesfor modulating the pixel registers 107. Each pixel register(bidirectional counter) 107 is then sent instructions indicating whichgroups to negate versus time to render a quasi-random, ideallyuncorrelated space-time code.

If desired, pseudo-random pixel group assignments can be used fortemporal multiplexing, employing a local ‘flutter pixel’, in thefollowing manner with the focal plane imaging apparatus 100 shown inFIG. 1A. First, the pixels 102 are pseudo-randomly selected and assignedto separate groups (e.g., four separate groups) in memory, such that allof the pixels 102 are pseudo-randomly assigned to the different groups.Second, a binary-valued (e.g., −1,1) pseudo-random aperture code isloaded from memory. Third, during readout of a single detector,pre-selected groups are negated (i.e., multiplied by −1) T times. Toensure accurate data value interpretation by the data acquisitionsystem, T may be an even number.

The processor 120 coupled to the DROIC 130 polls the registers 107 at areadout rate slower than the pseudo-random modulation rate (e.g., thereadout rate may be on the order of kilohertz, whereas the pseudo-randommodulation rate may be on the order of Megahertz). If each pixelincludes multiple digital registers 107—e.g., one to collect themodulated image data and another one to collect an unmodulated image ofthe scene—the processor 120 may read data from one, some, or all of theregisters 107 in each pixel 102.

The processor 120 processes the digital numbers read from the registers107 based on the pseudo-random modulation applied to each pixel 102 orgroup of pixels 102 in the DROIC 130. For example, the processor 120 maydiscriminate at least one temporal frequency in the scene based on thepseudo-randomly modulated representation of the scene read from theregisters 107. In some cases, the processor 120 applies a compressivesampling/sensing algorithm for calculating and outputting areconstructed image of the scene based on a pseudo-random modulation anda stored cumulative value of each of the digital registers, as discussedbelow in the section titled “Compressive Sampling Algorithms andTransient Image Reconstruction.”

The DROIC 130 can also be used to implement wide variety of digitallogic functions (see, e.g., FIG. 9 and its corresponding descriptionbelow). Under control of the processor 120, for example, the registers107 and transfer lines 104 can implement a combination of logic gates(e.g., AND, OR, NOT, NAND, NOR, XOR, XNOR) by transferring data betweenregisters 107 and incrementing and/or decrementing the counts in theregisters 107 as desired. The processor 120 and/or the local controllogic 113 can also implement logical operations on the digital numberstored in the register 107 by incrementing, decrementing, or negatingthe digital number and/or shifting it to another register 107 in thesame pixel 102 or a different pixel 102. These changes may be based on aprevious digital number or a previous state of the corresponding digitalregister 107 and/or a group assignment of the register 107 issued by theelectronic circuitry. The local control logic 113 may also set theregister 107 to a new/subsequent register state based on the previousregister state and/or the ADC 105 output. These data transfer,increment, decrement, and negate operations can be used to realizedesired Boolean functions of one or more input signals. They can also beused to perform on-chip filtering, motion compensation, etc. of thedetected image data.

FIG. 1D shows a digital pixel 102 in which one or more control registers141 are operably coupled through an interface with counter 107. Thecontrol registers 141 may be part of a distributed control patterngenerator, as discussed in further detail below. If the modulationsignal is generated off-chip by off-chip electronics 143 and transferredto the control registers 141 via a communications read-in interface 142,then that modulation can be used to enable and disable the counter andto control its count direction. The modulation rate may be limited tothe rate at which new register states can be transferred from theoff-chip electronics via interface 142 to the one or more controlregisters 141.

The off-chip electronics 143 may comprise a pseudorandom numbergenerator, such as the linear feedback shift register illustrated inFIG. 11. As shown in FIG. 1D, the control registers 141 can also beoperably coupled to other pixels 102 (e.g., neighboring pixels,optionally in one or more orthogonal directions within the array) in theDROIC 130, e.g., to receive configuration information via an on-chipdistributed control pattern generator (discussed further below). Thedistributed control pattern generator comprises non-local circuitry thatis distributed throughout the pixel array, utilizing local componentlogic blocks that reside within each pixel. The distributed controlpattern generator may additionally rely on additional circuitry thatresides at the periphery of the pixel array or off-chip.

A focal plane imaging apparatus may include additional local circuitryto impart a modulation that includes frequency information that can befaster than the rate at which data may be read onto the chip viainterface 142. FIG. 1E shows an example embodiment in which themodulation data is transferred from the off-chip electronics 143 to acontrol register ring buffer 148. Within the ring buffer 148 are storageregisters 144 a-d and a control register 145. The control register 145is operably coupled through an interface with the counter 107. Datawithin the ring buffer 148 may be advanced at a rate faster than thedata read-in rate. However, the number of states in the pseudorandomsequence is limited by the number of available registers within thepixel, thus limiting the scalability of the implementation to tightpixel pitches. As shown in FIG. 1E, the control register ring buffer 148can also be operably coupled to other pixels 102 (e.g., neighboringpixels, optionally in one or more orthogonal directions within thearray) in the DROIC 130, e.g., to receive configuration information viaan on-chip distributed control pattern generator (discussed furtherbelow).

In an alternate embodiment, illustrated in FIG. 1F, configuration datafrom the off-chip electronics 143 is transferred via a read-in interface142 to a configuration register 157 that resides in local logic block158. The configuration register 157 may have any number of configurationbits. A comparator 159 coupled to the configuration register 157compares the configuration bit(s) to a control condition 149 that can beupdated in real time by the control electronics 143 at a rate fasterthan the read-in frame rate. As shown in FIGS. 1F-1I, the local logicblock 158 can also be operably coupled to local logic blocks 158 inother pixels 102 (e.g., neighboring pixels, optionally in one or moreorthogonal directions within the array) in the DROIC 130, e.g., toreceive configuration information via an on-chip distributed controlpattern generator (discussed further below).

In some examples, the configuration register 157 assigns a pixel 102 toa group (e.g., a group of pixels with the same modulation pattern, orwhose modulation will be changed in a common manner and/orconcurrently), and the control condition is used to identify which pixelgroup is to have a particular setting. For example, as discussed furtherbelow, FIG. 8 illustrates an example of a two-bit comparator, in whichthe configuration memory 815 corresponds to the configuration register3202. The number of pixel groups is given by two raised to the power ofthe number of configuration register bits. Thus for the example two-bitimplementation, there are four available pixel groups.

The quality of achievable reconstruction of an acquired image can beimproved by reducing the mask correlation between pixels (see, e.g., thediscussion of FIGS. 13A through 13H below). One way to reduce maskcorrelation between pixels is to include a pseudorandom patterngenerator in each pixel instead of applying modulation to pixel groups.One approach to implementing a pseudorandom pattern generator in eachpixel is to use an LFSR in each pixel.

FIG. 1G shows an embodiment in which a linear feedback shift register161 is used, in place of the configuration register 157 of FIG. 1G, toprovide a group assignment. FIG. 1H shows an embodiment in which alinear feedback shift register 161 is instantiated within the pixellogic block 158 and selected bits 163 are operably coupled to thecounter. However, placing an entire linear feedback shift registerwithin a pixel requires a considerable amount of pixel area and limitsscalability. As such, some embodiments do not include an entire LFSRwithin a single pixel. Instead of mapping a single LFSR to a singlepixel, these embodiments use a single LFSR to control several pixels ofthe DROIC, e.g., with one or more LFSRs controlling all of the pixels ina given DROIC. These LFSRs can reside within the DROIC, or can resideoff-chip and communicate with the appropriate pixels (e.g., by way ofcommunications interface 142). The transfer, storage and/or logicalmanipulation of LFSR data can be achieved by way of an on-chipdistributed control pattern generator, as discussed below.

Distributed Control Pattern Generator

FIG. 2A illustrates a distributed control pattern generator 200 thatgenerates and distributes pseudorandom modulation without a separatepseudorandom pattern generator in each pixel. More specifically, FIG. 2Ashows a plurality of pixel control registers 241 connected to each othervia an inter-pixel data transfer network 204. Together, the pixelcontrol registers 241 and the pixel data transfer network 204 form thedistributed control pattern generator circuit 200.

The data transfer network 204 can include electrical interconnect forpassing signals between pixels of the focal plane imaging apparatusand/or for passing signals onto/off chip, and electronic logiccomponents/circuitry for manipulating signals that are transferredbetween pixels. The data transfer network 204 and pixel controlregisters 241 may optionally form or include one or more LFSRs. The setof pixel control registers 241 are loaded either by way of an off-chipcommunication interface 142 (as shown in FIGS. 1D-1H) or with valuesresiding elsewhere in the pixel 202. As discussed below with referenceto FIG. 7, data from a digital register (e.g., digital register 707 inFIG. 7) can be input to a logic block within the pixel (e.g., logicblock 713 in FIG. 7), in which can reside a control register 241. Inoperation, a pattern is first loaded from off-chip via interface 142,and during operation the pattern information is transferred from pixel202 to neighboring pixel 202 using the inter-pixel data transfer network204 to create a modulated pattern. Note that each register 241 takesamong its inputs a set of values from other registers 241.

An exemplary distributed control pattern generator 200′, with aninter-pixel data transfer network, is illustrated schematically in FIG.2B. This distributed control pattern generator 200′ includes adestination pixel 202′ with a destination register 225 that is populatedwith values from a selected source register 211. A multiplexor 220 inthe destination pixel 202′ selects between the selected register 211 andunselected registers 210 based on a selector signal 215. The selectorsignal 215 may come from other control registers 241, from on-chip logic(e.g., 113 in FIG. 1A), or from an off-chip signal (e.g., read-ininterface 142 in FIGS. 1D-1I).

FIG. 2C shows another embodiment of a distributed control patterngenerator 200″, in which an exclusive OR (XOR) gate 214 is added to atleast one pixel 202″ in the inter-pixel transfer network 204 so that oneof the input options selected by selector signal 215 is the exclusive ORof outputs from control registers 212. This allows the destinationcontrol register 225 to receive either a value from a source controlregister 210 or to receive the XOR product of multiple control registers212. By configuring the selector signals for each instance of themultiplexor 220, local implementations of linear feedback shift registercircuits (such as is illustrated in FIG. 11) may be created in adistributed manner throughout the pixel array. In place of the XOR gate214, other logic configurations may be used to form a variety of statemachine types known to the art to generate a desired modulation pattern,where the modulation pattern state machine is similarly distributedthrough the pixel array.

FIGS. 2D and 2E illustrate embodiments of a distributed control patterngenerator 200′″ and 200″″, in which the pixel array is implemented witha combination of direct inter-pixel transfer connections 204 a andXOR-gate transfer connections 204 b, respectively. For the directconnections 204 a in FIG. 2D, data is transferred directly from a sourceregister 210 to a destination register 225 in a destination pixel. Forthe XOR-gate transfer connections 204 b in FIG. 2E, the XOR product ofsource registers 212 is computed by XOR gate 214 in a destination pixel202″″ and is received and stored by destination register 225 in thedestination pixel 202″″. Through selection of interconnectivity ofregisters and placement of instances of 204 a and 204 b, linear feedbackshift register structures and other structured pattern generationstructures may be implemented. For some embodiments, some instances ofsource register 210 or 212 may be replaced with hardwired zero or onevalues. In place of the XOR gate 214, other logic configurations may beused to form a variety of state machine types known to the art togenerate a desired modulation pattern, where the modulation patternstate machine is similarly distributed through the pixel array.

Pixel Circuit Configurations

FIG. 3 is a schematic showing a pixel circuit with a pulse modulator 305and a bidirectional counter 307 suitable for use in the focal planeimaging apparatus of FIGS. 1A and 1D-1I. The pixel circuit includes aphotodetector 301 for receiving an input photocurrent, and apreamplifier 303 to amplify the current signal that is supplied to thepulse modulator 305 for analog-to-digital conversion. The converteddigital signal (a digital representation of the input photocurrent) isthen routed to the bidirectional counter 307 and stored. Gating signals,supplied via computation support logic 309, selectively apply processingoperations to particular pixels via the pulse modulator 305 and/or thebidirectional counter 307. The pixel selection can be based on multiplecriteria, such as peripheral selection signals, in-pixel configurationbits, counter state tests (e.g., thresholding and absolute value),and/or local control or modulation signals 311. For example, the gatingsignals output by the computation support logic 309 can be “countdependent,” based on a value stored within the bidirectional counter307. As shown in FIG. 3, each bidirectional counter can be electricallycoupled to an adjacent pixel and configured for “nearest neighbor”communication therewith.

In addition to incrementing proportionately to photocurrent, asdescribed above with reference to FIG. 1A, each pixel circuit can beimplemented to allow a wider variety of functions of photocurrent andprevious state. For example, a pixel circuit may include a bidirectionalcounter (i.e., a count up/down counter) like the one in FIG. 4A and/or anegate counter like the one in FIG. 4B. A bidirectional counter has adirection pin that sets whether it will increment or decrement on eachclock cycle. A negate counter is a counter that only increments, butthat also can negate its value. The same result as decrementing on abidirectional counter can be achieved using a negate counter bynegating, incrementing, and negating again. To illustrate, taking a fourinput clock cycle starting at a value of “3” and counting down (i.e.,using a bidirectional counter), the count would be “2, 1, 0, −1” (i.e.,a result of “−1”). To obtain the same result using a negate-typecounter, the 3 is first negated (to “−3”), the count proceeds as “−2−1 01,” and the result is negated to get “−1.” In addition to incrementingor decrementing by a value of “1,” other values can also be added orsubtracted (e.g., 2, 4, etc.). In some embodiments, a value of theincrement or decrement is determined/dictated by one or locally storedbits (e.g., in the configuration memory).

FIG. 5 is a schematic showing a pixel circuit with local gainconfigurability, according to some embodiments. In order to allow eachpixel to independently process the incoming photocurrent (generated byphotodetector 401) according to its own specified transfer function,pixel configurability can be implemented. As shown in FIG. 5, the gainof the ADC 505 (e.g., the gain of a current-to-frequency converter) canbe set by a digital input to that module. This digital input may beconnected to a local configuration memory 511 (as shown in FIG. 5), aregister bank, or any other digital bus. Alternatively or in addition,the modulation of storage in one or more counters can also be set by adigital input. In some embodiments, a gain set function can refer to ananalog conversion gain applied to the photodiode 501. For example, thegain set function can be implemented by performing a digital-to-analogconversion of conversion data (e.g., in the local configuration memory)and subsequent modulation of the photodiode gain.

Another way in which pixels may be independently configured to implementtheir own transfer functions is through the use of an enable pin on thecounter/register structure. FIG. 6 is a schematic showing acounter/register block with a clock, a mode selection bus, and an enablepin, according to some embodiments. The mode selection bus can determinethe operation to be performed at each clock cycle, and the enable pincan determine whether the circuit heeds the clock. The mode select caninclude a wide variety of operations that include increment, decrement,negation, shifting, etc.

FIG. 7 is a schematic showing a pixel circuit with a photodetector 701,an ADC 705, logic module 713, and a digital register 707, according tosome embodiments. In addition to performing a raw transfer of the ADC705 output to the digital register 707, the logic module 713 can set anew register state of the digital register 707. The new register statemay be a function of a previous state of the digital register 707 andthe ADC 705 output. During operation, photons incident on thephotodetector 701 are converted by the photodetector 701 intophotocurrent that is electrically routed to the ADC 705 for conversioninto a digital signal (a digital representation of the photocurrent).After the analog-to-digital conversion, the digital signal is thenrouted to a logic module 713 for manipulation (e.g., via a logicoperation) prior to arriving at the digital register 707 where thedigital representation (e.g., as modified by logic module 713) may ormay not be stored, a count may be incremented or decremented, etc. Inother words, the logic module 713 sets a register state (e.g., countdirection, enable/disable, etc.) of the digital register 707. In someembodiments, the output of the digital register 707 can be electricallytied to the logic module 713, for example such that a “new” registerstate of the digital register 707 is set to a designated function of aprevious state of the digital register 707 and the ADC 705 output. Insome embodiments, a type of logic function employed by the logic module713 is dependent upon an output state of the digital register 707.

FIG. 8 is a schematic showing a pixel circuit with a photodetector 801,a preamplifier 803, a pulse frequency modulator 805, local control logic813 enabling a pseudorandom modulation sequence, and a bidirectionalcounter 807, according to some embodiments. A photocurrent (“I_(photo)”)proportional to an incident photon flux is induced in photodetector 801,and amplified by preamplifier 803. The amplified analog photocurrentsignal is converted to a “pulse train” digital signal by the pulsefrequency modulator 805. The digital signal is routed to a clock pin ofthe counter 807, clocking the digital counter 807 (for example, byincrementing with each pulse). One or more pseudorandom modulationsequences are produced by the local control logic module 813, which iselectrically coupled to an enable pin and an up/down (i.e., “modeselect”) pin of the counter 807, and the modulated imaging data is readout from the counter 807.

Pixel Group Assignments

In some embodiments, individual pixels are assigned to one of aplurality of “groups,” with the group assignments stored in the mapregister 140 shown in FIG. 1A. Pixel group membership can be used, forexample, to determine whether or not to increment or negate acorresponding counter of the pixel over a predetermined segment of time.Group assignments for individual pixels are pre-set within aconfiguration memory. FIG. 9 is a schematic showing a digital logiccircuit in which a two-bit pixel group assignment is used to assign apixel to one of four different groups. A value in the configurationmemory 915 is compared to a value on a two-bit control bus (controlsignals “CLKCTRL(0)” and “CLKCTRL(1)”), and the enable signal is onlyhigh when the value on the control bus matches the value in theconfiguration memory (i.e., CLKCTRL=CLKPROG). The enable in FIG. 9 canbe used, for example, as an enable input in FIG. 6. The approach shownin FIG. 9 can be extended to an arbitrary number of bits.

FIG. 10 is a schematic showing a linear feedback shift register (“LFSR”)applying pseudorandom pixel group assignment to a plurality of pixels,according to some embodiments. Each pixel of the focal plane imagingapparatus can be assigned one or more bits of the LFSR 1021, and LFSRscan be local to one pixel or to a neighborhood of pixels. The LFSR 1021may be “seeded,” for example with bits from the resident image in thecounter/register or with a pre-loaded pattern, allowing for a seed thatvaries across the array. The LFSR 1021 can be clocked at a rate muchfaster than the integration time, so multiple LFSR clocks can be placedbetween integrations, thereby increasing spatial decorrelation.

In some embodiments, the pixel architecture contains a 14-bitconfiguration register where a pre-loaded binary-valued mask is loadedinto memory and a map register (e.g., map register 140 in FIG. 1A)stores four separate group assignments. The group assignments areassigned via LFSR 1123 (see FIG. 11) utilizing the least-significant bitand most-significant bit of the 14-bit map register to assign pixels tofour separate pixel groups. Note that LFSRs can be used to producepseudo-random sequences. A bidirectional counter is then sentinstructions indicating which groups to negate versus time to render aspatially and temporally quasi-random space-time code. To generate thegroup assignments pseudorandomly, a pseudorandom number generator isemployed. FIG. 11 shows an LFSR that cycles through a pseudorandomsequence of 2N−1 numbers, where N is the number of bits in the LFSR(e.g., see “Bit0,” “Bit1,” and “Bitn” in FIG. 10). Pseudorandom groupassignment can be used to map a set of enable pins to a set of LFSRbits. The time-varying group assignments can have a distribution ofvalues across the plurality of digital registers.

FIG. 12 shows an embodiment in which every 4-by-4 neighborhood of pixelsincludes a 16-bit linear feedback shift register as illustrated in FIG.11, using the register connectivity approach presented in FIGS. 2C and2D. In this case, the LFSR bit position shown in 1227 is tiled across alarger array of pixels 1225. While the pixel array has a repeating4-by-4 pattern of interconnectivity, the entire array may be loaded witha seed matrix so that different 4-by-4 neighborhoods receive differentseed values. The linear feedback shift register may be clocked anynumber of times between use of its value, this allowing for furtherreduction of mask correlation.

Hence, using the architecture shown in FIG. 12, there is one LFSR per4×4 neighborhood, and this neighborhood structure is repeated 64×64times in a 256×256 detector array. Note that in this case multiple LFSRclocks can be used to avoid horizontal or vertical spatial correlation.

In some embodiments, horizontal or vertical correlations are desirablein that they implement a spatiotemporal gradient filter. For the exampleshown in FIG. 11, one LFSR clock between integrations will implement ahorizontal correlation, two clocks will implement a verticalcorrelation, and three clocks will implement a diagonal correlation.Sixteen clocks will implement no correlation. By modifying the size ofthe LFSR and the assignment of LFSR bits to enable signals, spatialcorrelation can be minimized or maximized according to the designintent.

FIGS. 13A and 13B are space-time per-pixel modulation correlationmatrices for pixel-group based encoding, according to some embodiments,and the space-time per-pixel modulation correlation matrix for a pixelarchitecture employing a local LFSR pseudorandom encoding (e.g., 4×4region of interest) 4,096 times across a 256×256 2D array, is shown inFIG. 13C. As shown in FIGS. 13A-13C, the local LFSR-based pseudorandomimplementation improves the space-time correlation, thereby yieldingimproved 3D (e.g., x, y, and t) transient target recovery from a 2Dspace-time encoded projection recorded at the base detector frame rate.FIG. 13D is a space-time per-pixel modulation correlation matrix forencoding using four pixel groups, according to some embodiments, and canbe contrasted with FIG. 13E, which shows a space-time per-pixelmodulation correlation matrix for “ideal” LFSR modulation withoff-diagonal mask correlation, according to some embodiments.

FIGS. 13F-13H shows that a notable reduction in mask correlation can beachieved by using a distributed LFSR approach 1300, as compared to thepreviously described approach using four pixel groups (e.g., asdiscussed with reference to FIG. 12 and as represented by FIG. 13F). Theplot in FIG. 13F (like FIG. 13D) shows the typical mask correlation forthe case in which four pixel groups defined by configuration registers(such as configuration register 157 in FIG. 1F) are populated by theoff-chip electronics (such as 143 in FIG. 1F) via an interface (such asread-in interface 142 in FIG. 1F) using bits 0 and 15 of a 16-bit linearfeedback shift register. When compared to the “ideal” mask correlation(e.g., as shown in FIG. 13E) that could be obtained if each pixel hadits own linear feedback shift register, the mask correlation is high,limiting the quality of achievable reconstruction.

System Operation

FIG. 14 is a flow chart that illustrates a process for focal planeimaging using a digital focal plane array (DFPA), according to someembodiments. Light reflected from a scene of interest is received, at1421, at an array of photodetectors 1401 each of which generates acorresponding photocurrent. The photocurrents are converted, at 1423,into a plurality of corresponding digital signals (each represented by adigital number) at a plurality of corresponding ADCs 1405. The gain ofeach of the ADCs 1405 may be pseudo-randomly modulated (at 1423A). Thedigital numbers are stored, at 1425, in a plurality of correspondingdigital registers 1407, each comprising an enable pin that may bepseudo-randomly modulated (at 1425A). The process shown in FIG. 14 canbe performed on a single integrated circuit (IC) chip.

In some embodiments, processes described herein further includecalculating, via a processor, a three-dimensional (x, y, t) image of thescene of interest based on the pseudo-random modulation and valuesstored in the plurality of digital registers.

FIG. 15 shows a series of images that have been reconstructed usinglocal, in-pixel pseudorandom time-varying modulation, according to someembodiments. A random temporal mask 1530 is employed, with an exemplarylocal in-pixel temporal gain modulation pattern shown at 1532, and aresulting output frame is shown at 1534. The reconstructed image isproduced at 15 x the frame rate, and fifteen image “slices,” acquiredwithin a single frame, are shown in the sequence of 1534 a to 1534 o. Insome embodiments, image reconstruction is accomplished using compressivesensing algorithms, as discussed below.

Compressive Sampling Algorithms and Transient Image Reconstruction

Compressive sampling algorithms can be applied to coded-aperturetemporal imaging to enable (x, y, t) data-cube estimation from alow-speed temporally encoded projection of transient targets.Compressive sensing enables data recovery at sub-Nyquist sampling rates.The sparsity matrix and the measurement matrix can be designed such thatthe matrix combination satisfies the restricted isometry property(“RIP”). Incoherence between the sparsity matrix and measurement matrixis an important metric. The combination of on-chip per-pixel modulationand algorithmic recovery from sub-Nyquist measurements enables 3D(x, y,t) data recovery from a 2D(x, y) space-time encoded measurement.

In some embodiments, a mathematical framework for 2D space-time encodingis defined as

$\begin{matrix}{{g_{i,j} = {{\sum\limits_{t}^{T}{M_{i,j,t}f_{i,j,t}}} + n_{i,j}}},} & (1)\end{matrix}$where f_(i,j,t) corresponds to the discretized form of the 3D (x,y,t)target vector at spatial indices (i,j) and at time t, M_(i,j,t)represents the per-pixel modulation at spatial indices (i,j) and at timet, n_(i,j) is the additive white Gaussian noise at spatial indices(i,j), and g is the measured data over all pixels (N_(R)×N_(C)) and overT sub-frames. The temporally-varying modulation at each pixel modulatesa moving scene at rates faster than the readout rate of the detector.The forward model in matrix notation is defined as:g=Mf+n,  (2)where f

^(N) _(R) ^(N) _(c) ^(Txl) is the discrete object vector, g

^(N) _(R) ^(N) _(c) ^(xl) is the measurement vector, M

^(N) _(R) ^(N) _(c) ^(N) _(R) ^(N) _(c) ^(T) is the binary-valued (−1,1or 1,0) time-varying per-pixel modulation matrix, and the noise vectoris represented as n

^(N) _(R) ^(N) _(c) ^(xl). The matrix M inherently includes the systemoptical blur and embeds the temporal downsampling operation of theslow-speed camera. Assuming a unitary M matrix, the transpose model isdefined as:f _(est) =M ^(T) g,  (3)where f_(est) is the linear estimate and M^(T) is also equivalent toM⁻¹. An adapted two-step iterative shrinkage/thresholding (TwIST) totalvariation (TV) minimization algorithm is used for data-cube (x,y,t)estimation from a single 2D (x,y) space-time encoded image of atransient scene. The least squares data-cube estimate, f_(est), providesa spatial (x,y) and temporal (t) representation of a scene of interest.Using TwIST, a convex objective function is minimized. TwIST is definedas:f*=arg_(f)min∥g−Mf∥ ₂ ²+τφ_(iTV)(f)′  (4)where f* is the TwIST data-cube estimate, τ is the regularizationscaling parameter (e.g., small values preserves edges and large valuesblur the data-cube estimate), and φ_(iTV) represents the isotropicdiscrete TV regularizer. The isotropic discrete TV regularizer in twodimensions is defined as

$\begin{matrix}{{{\varphi_{iTV}(f)} = {\sum\limits_{t}^{\;}\sqrt{\left( {\Delta_{t}^{h}f} \right)^{2} + \left( {\Delta_{t}^{v}f} \right)^{2}}}},} & (5)\end{matrix}$where Δ^(h) _(t)f and Δ^(v) _(t)f are gradient (i.e., difference)operators along the horizontal and vertical spatial dimensions of theimage. The TV regularizer is further defined as:φ_(iTV,3D)(f)=Σ_(t)Σ_(t,j)√{square root over ((f _(t+1,j,k) −f_(i,j,t))²+(f _(i,j+1,k) −f _(i,j,t))²)}.  (6)The nonlinear convex optimization framework can apply a gradientconstraint to each temporal slice of the estimated space-time data cubeindependently.

FIG. 16A is a spatially extended baseline image of three transienttargets (bullets), according to some embodiments. FIG. 16B is an exampleframe of a spatially randomized temporally modulated coded aperture usedfor imaging the baseline image of FIG. 16A. The transient target of FIG.16A was imaged by a pixel array where each pixel is temporally modulatedas illustrated temporally (via an LFSR) in FIG. 10 and spatially inFigure FIG. 16B. The raw modulated pixel output image at the detector,taken at the base readout frame rate, is shown in FIG. 16C. FIG. 16D isa series of images, 1634 a-1634 o, comprising reconstructed datayielding 15 time slices per frame acquired while imaging the transienttarget of FIG. 16A. Since the modulation, shown in FIG. 16B, occurs muchfaster than the readout frame rate, high-speed data can be reconstructedusing compressive sensing algorithms as shown in FIG. 16D. The detectorarchitecture preserves signal flux (i.e., 3 dB increase) by employing a−1,1 binary-valued mask instead of a 1,0 mask. FIG. 16E is a plot ofpeak signal-to-noise ratio (“PSNR”) values comparing the baseline(x,y,t) target data of FIG. 16A with the (x,y,t) data-cube estimatesversus sub-frame number.

Conventional, space-time per-pixel modulation, and space-time globalmodulation data captured at the base detector frame rate can be comparedin FIGS. 17A-C.

FIG. 17A is a simulated image of three transient targets of FIG. 16Arecorded using conventional modulation at a base readout frame rate,according to some embodiments. As shown in FIG. 17A, conventionalmeasurements cannot recover transient target features from the streak.

FIG. 17B is a simulated image of the three transient targets of FIG. 16Ausing on-chip per-pixel modulation at a base readout frame rate,according to some embodiments. As shown in FIG. 17B, on-chip per-pixelmodulation enables snapshot multi-target and multi-frequency transientfeature extraction.

FIG. 17C is a simulated image of the three transient targets of FIG. 16Ausing on-chip global modulation at a base readout frame rate, accordingto some embodiments. As shown in FIG. 17C, on-chip global modulationenables single frequency extraction.

As shown in FIGS. 17A-C, per-pixel modulation impacts the ability for3D(x,y,t) target recovery. The per-pixel modulation manifests itself inat least the following two manners, depending upon the pixelimplementation at the configuration register or via a local LFSRimplementation, to generate a true pseudorandom space-time code.

Digital-domain per-pixel modulation can be modeled as

$\begin{matrix}{{{M\left( {x,y} \right)} = {\sum\limits_{i}^{\;}{\sum\limits_{j}^{\;}{M_{i,j}{{rect}\left( {\frac{x - {i\;\Delta_{m}}}{\Delta_{m}},\frac{y - {j\;\Delta_{m}}}{\Delta_{m}}} \right)}}}}},} & (7)\end{matrix}$where M_(jj) represents the amplitude value at the (i,j)^(th) positionin the aperture code and Δ_(m) is the aperture-code pitch. Consideringthe pixel architectures in FIG. 5 and FIG. 9, N_(R)×N_(C) pixels arepseudorandomly assigned to four different pixel groups. A binary-valued(i.e., −1,1) space-time modulation is loaded from memory. Finally,during a single detector readout, pre-selected groups are negated (i.e.,multiply by −1) T times. A final detector in-pixel modulation can bemodeled as:

$\begin{matrix}{{g_{n} = {{\sum\limits_{x = 1}^{T/2}{f_{i = {{2x} - 1}}\left\lbrack {\prod\limits_{j = 1}^{T}M_{j}} \right\rbrack}} - {\sum\limits_{x = 1}^{T/2}{f_{i - {2x}}\left\lbrack {\prod\limits_{j = 1}^{T}M_{j}} \right\rbrack}}}},} & (8)\end{matrix}$where T represents the number of sub-frames averaged in the detectorreadout period and recall that f is the transient object. The space-timecorrelation matrix for the modulation defined with Eqn.8 is shown inFIG. 13A. A more direct per-pixel modulation is denoted as

$\begin{matrix}{g_{n} = {\sum\limits_{t}^{T}{M_{t}f_{t}}}} & (9)\end{matrix}$and the space-time correlation matrix, maintaining the group assignmentimplementation is shown in FIG. 13B.

FIGS. 18A and 19 are annotated image frames showing pixel groupassignments, according to some embodiments. Each pixel in the image isassigned to a group or region of interest (e.g., Groups “0” and “1” aredefined in FIG. 18A, and Groups “0” through “2” are defined in FIG. 19).FIGS. 18B and 18C are image frames created using context-relevant signalprocessing (rotation compensation) within each group shown in FIG. 18A.

CONCLUSION

While various inventive embodiments have been described and illustratedherein, those of ordinary skill in the art will readily envision avariety of other means and/or structures for performing the functionand/or obtaining the results and/or one or more of the advantagesdescribed herein, and each of such variations and/or modifications isdeemed to be within the scope of the inventive embodiments describedherein. More generally, those skilled in the art will readily appreciatethat all parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and that the actual parameters,dimensions, materials, and/or configurations will depend upon thespecific application or applications for which the inventive teachingsis/are used. Those skilled in the art will recognize or be able toascertain, using no more than routine experimentation, many equivalentsto the specific inventive embodiments described herein. It is,therefore, to be understood that the foregoing embodiments are presentedby way of example only and that, within the scope of the appended claimsand equivalents thereto, inventive embodiments may be practicedotherwise than as specifically described and claimed. Inventiveembodiments of the present disclosure are directed to each individualfeature, system, article, material, kit, and/or method described herein.In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the inventive scope of the present disclosure.

The above-described embodiments can be implemented in any of numerousways. For example, embodiments of designing and making the technologydisclosed herein may be implemented using hardware, software or acombination thereof. When implemented in software, the software code canbe executed on any suitable processor or collection of processors,whether provided in a single computer or distributed among multiplecomputers.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device not generally regarded as acomputer but with suitable processing capabilities, including a PersonalDigital Assistant (PDA), a smart phone or any other suitable portable orfixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.

Such computers may be interconnected by one or more networks in anysuitable form, including a local area network or a wide area network,such as an enterprise network, and intelligent network (IN) or theInternet. Such networks may be based on any suitable technology and mayoperate according to any suitable protocol and may include wirelessnetworks, wired networks or fiber optic networks.

The various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, various inventive concepts may be embodied as acomputer readable storage medium (or multiple computer readable storagemedia) (e.g., a computer memory, one or more floppy discs, compactdiscs, optical discs, magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other non-transitory medium or tangible computer storagemedium) encoded with one or more programs that, when executed on one ormore computers or other processors, perform methods that implement thevarious embodiments of the invention discussed above. The computerreadable medium or media can be transportable, such that the program orprograms stored thereon can be loaded onto one or more differentcomputers or other processors to implement various aspects of thepresent invention as discussed above.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of embodiments as discussedabove. Additionally, it should be appreciated that according to oneaspect, one or more computer programs that when executed perform methodsof the present invention need not reside on a single computer orprocessor, but may be distributed in a modular fashion amongst a numberof different computers or processors to implement various aspects of thepresent invention.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconvey relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Also, various inventive concepts may be embodied as one or more methods,of which an example has been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of.” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03.

The invention claimed is:
 1. A focal plane imaging apparatus comprising: a photodetector array to convert light that is scattered and/or reflected from a scene into analog signals; analog to digital converters (ADCs), electrically coupled to the photodetector array, to convert the analog signals into digital signals; digital registers, electrically coupled to the ADCs, to store a two-dimensional space-time encoded measurement produced by modulating the digital signals with pseudorandom modulation; and a processor, operably coupled to the digital registers, to generate a three-dimensional representation of the scene from the two-dimensional space-time encoded measurement.
 2. The focal plane imaging apparatus of claim 1, wherein the photodetector array has a readout rate slower than a modulation rate of the pseudorandom modulation.
 3. The focal plane imaging apparatus of claim 1, wherein the processor is configured to generate the three-dimensional representation of the scene by applying a sparsity constraint to the two-dimensional space-time encoded measurement.
 4. The focal plane imaging apparatus of claim 1, wherein the processor is configured to generate the three-dimensional representation of the scene by applying a two-step iterative shrinkage/thresholding (TwIST) total variation (TV) minimization process to the two-dimensional space-time encoded measurement.
 5. The focal plane imaging apparatus of claim 1, wherein the processor is configured to extract at least one target from the three-dimensional representation of the scene.
 6. The focal plane imaging apparatus of claim 1, wherein the processor is configured to extract at least one frequency from the three-dimensional representation of the scene.
 7. The focal plane imaging apparatus of claim 1, further comprising: circuitry, operably coupled to the ADCs and/or the digital registers, to generate the pseudorandom modulation.
 8. The focal plane imaging apparatus of claim 7, wherein the circuitry is configured to modulate different digital signals with different pseudorandom modulation signals.
 9. The focal plane imaging apparatus of claim 7, wherein the circuitry is configured to modulate a gain of the ADCs with the pseudorandom modulation.
 10. The focal plane imaging apparatus of claim 7, wherein the circuitry is configured to modulate the digital registers with the pseudorandom modulation.
 11. A method of focal plane imaging, the method comprising: imaging a scene onto a photodetector array; converting a photocurrent, generated by the photodetector array, into digital signals using analog-to-digital converters (ADCs); storing, with digital registers operably coupled to the ADCs, a two-dimensional space-time encoded measurement produced by modulating the digital signals with pseudorandom modulation; and generating, with a processor operably coupled to the plurality of digital registers, a three-dimensional representation of the scene from the two-dimensional space-time encoded measurement.
 12. The method of claim 11, wherein modulating the digital signals with the pseudorandom modulation occurs at a rate faster than a readout rate of the photodetector array.
 13. The method of claim 11, wherein modulating the digital signals with the pseudorandom modulation comprises modulating different digital signals with different pseudorandom modulation signals.
 14. The method of claim 11, wherein modulating the digital signals with the pseudorandom modulation comprises modulating a gain of at least one of the ADCs with the pseudorandom modulation.
 15. The method of claim 11, wherein modulating the digital signals with the pseudorandom modulation comprises modulating at least one of the digital registers with the pseudorandom modulation.
 16. The method of claim 11, wherein generating the three-dimensional representation of the scene comprises applying a sparsity constraint to the two-dimensional space-time encoded measurement.
 17. The method of claim 11, wherein generating the three-dimensional representation of the scene comprises applying a two-step iterative shrinkage/thresholding (TwIST) total variation (TV) minimization process to the two-dimensional space-time encoded measurement.
 18. The method of claim 11, further comprising: extracting at least one target from the three-dimensional representation of the scene.
 19. The method of claim 11, further comprising: extracting at least one frequency from the three-dimensional representation of the scene. 